{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(realtime-monitor-drift-tutor)=\n",
    "# Model monitoring tutorial\n",
    "\n",
    "This tutorial illustrates the basic model monitoring capabilities of MLRun: deploying a model to a live endpoint (model monitoring endpoint); deploying two model-monitoring applications, one of them based on the Evidently service / OSS; configuring alerts on data-drift; exercising the deployment with invocations, and viewing the calculated drift results and the UI feedback, and the triggered alerts.\n",
    "\n",
    "Make sure you went over the basics in MLRun [**Quick Start Tutorial**](01-mlrun-basics.ipynb).\n",
    "\n",
    "In this tutorial:\n",
    "- [Preparations](#preparations)\n",
    "- [Invoke the model](#invoke-the-model)\n",
    "- [Register and deploy model-monitoring apps](#register-and-deploy-the-model-monitoring-app)\n",
    "- [Configure alerts](#configure-alerts)\n",
    "- [Generate traffic against the serving function model](#generate-traffic-against-the-serving-function-model)\n",
    "- [View model monitoring artifacts and drift in the UI](#view-model-monitoring-artifacts-and-drift-in-the-ui)\n",
    "- [View model monitoring artifacts and drift in Grafana](#view-model-monitoring-artifacts-and-drift-in-grafana)\n",
    "- [Batch infer model-monitoring](#batch-infer-model-monitoring)\n",
    "\n",
    "**See also**\n",
    "- {ref}`monitoring-models`\n",
    "- {ref}`Model monitoring architecture<model-monitoring-des>`\n",
    "\n",
    "## Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import mlrun and Evidently. Then install the Evidently package (restart the kernel after install). \n",
    "```\n",
    "{admonition}Note\n",
    "If Evidently is already installed, comment out the install command. If not, run the Evidently install command once, comment it out, and then restart the notebook.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: evidently==0.7.5 in /User/.conda/envs/daniel311/lib/python3.11/site-packages (0.7.5)\n",
      "Requirement already satisfied: faker>=5.0.0 in /User/.conda/envs/daniel311/lib/python3.11/site-packages (from polyfactory>=2.6.3->litestar>=2.8.3->evidently==0.7.5) (37.8.0)\n",
      "Requirement already satisfied: six>=1.5 in /User/.conda/envs/daniel311/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas>=1.3.5->pandas[parquet]>=1.3.5->evidently==0.7.5) (1.17.0)\n"
     ]
    }
   ],
   "source": [
    "import mlrun\n",
    "import os\n",
    "\n",
    "# Install and use a version of evidently that was tested to be working with mlrun\n",
    "from mlrun.model_monitoring.applications.evidently import SUPPORTED_EVIDENTLY_VERSION\n",
    "\n",
    "# run this only once (restart the notebook after the install !!!)\n",
    "!pip install \"evidently=={SUPPORTED_EVIDENTLY_VERSION}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create your MLRun project.\n",
    "```{admonition} Note\n",
    "If you want to use S3 storage, see [Data store: S3](../store/datastore.md#s3).\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-09-21 09:36:48,831 [info] Project loaded successfully: {\"project_name\":\"tutorial-admin\"}\n"
     ]
    }
   ],
   "source": [
    "project = mlrun.get_or_create_project(\n",
    "    \"tutorial\", context=\"./\", user_project=True, allow_cross_project=True\n",
    ")\n",
    "\n",
    "# Backward compatibility: Support both S3_ENDPOINT_URL (deprecated) and AWS_ENDPOINT_URL_S3\n",
    "# TODO: Remove this in 1.12.0\n",
    "aws_url = os.environ.get(\"AWS_ENDPOINT_URL_S3\") or os.environ.get(\"S3_ENDPOINT_URL\")\n",
    "if aws_url:\n",
    "    os.environ[\"AWS_ENDPOINT_URL_S3\"] = aws_url\n",
    "    project.set_secrets({\"AWS_ENDPOINT_URL_S3\": aws_url})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set datastore profiles \n",
    "\n",
    " A [datastore profile](https://docs.mlrun.org/en/stable/store/datastore.html#data-store-profiles) holds all the information required to address an external data source, including credentials. Before you enable model monitoring, you must configure datastore profiles for TSDB and streaming platforms:\n",
    "- TSDB (time series database) stores, retrieves, and analyzes time-stamped data, such as model predictions statistics or monitoring application metrics.\n",
    "- Streaming platforms are used to deliver data continuously and efficiently. In the model monitoring infrastructure, they are used for the data flow between Nuclio functions, supporting real-time processing and enabling event-driven operations.\n",
    "\n",
    "Model monitoring supports [Kafka](https://docs.mlrun.org/en/stable/api/mlrun.datastore/index.html#mlrun.datastore.datastore_profile.DatastoreProfileKafkaSource) and [V3IO](https://docs.mlrun.org/en/stable/api/mlrun.datastore/index.html#mlrun.datastore.datastore_profile.DatastoreProfileV3io) as streaming platforms, and [TDEngine](https://docs.mlrun.org/en/stable/api/mlrun.datastore/index.html#mlrun.datastore.datastore_profile.TDEngineDatastoreProfile) and [V3IO](https://docs.mlrun.org/en/stable/api/mlrun.datastore/index.html#mlrun.datastore.datastore_profile.DatastoreProfileV3io) as TSDB platforms. \n",
    "\n",
    "After you configure and save the datastore profiles under your project, you need to call the `set_model_monitoring_credentials` API to register them with the model monitoring resources. (In this tutorial the `set_model_monitoring_credentials` API is wrapped under the helper `enable_model_monitoring` that is in the `src.model_monitoring_utils`.) See [set_model_monitoring_credentials](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.set_model_monitoring_credentials) for more details.\n",
    "\n",
    "### Enable model monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.model_monitoring_utils import enable_model_monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    ":::{Admonition} Note\n",
    "If this project was running with model monitoring enabled in a pre-1.8.0 version, disable the old model monitoring to update the configurations:\n",
    "```\n",
    "project.disable_model_monitoring(delete_stream_function=True)\n",
    "enable_model_monitoring(project=project)\n",
    "```\n",
    ":::\n",
    "### Log the model artifact\n",
    "\n",
    "This tutorial uses an existing MLRun dataset for training. You just need to specify it. \n",
    "Then you can log the model. See the parameter descriptions in [`log_model`](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.log_model). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-09-21 09:36:49,106 [info] Model monitoring credentials were set successfully. Please keep in mind that if you already had model monitoring functions / model monitoring infra / tracked model server deployed on your project, you will need to redeploy them. For redeploying the model monitoring infra, first disable it using `project.disable_model_monitoring()` and then enable it using `project.enable_model_monitoring()`.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.projects.project.MlrunProject at 0x7fed5eb8fad0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enable_model_monitoring(project=project)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the training set\n",
    "import pandas as pd\n",
    "\n",
    "train_set = pd.read_csv(\n",
    "    \"https://s3.us-east-1.wasabisys.com/iguazio/data/iris/iris_dataset.csv\"\n",
    ")\n",
    "\n",
    "# Log the model\n",
    "model_name = \"RandomForestClassifier\"\n",
    "model_artifact = project.log_model(\n",
    "    model_name,\n",
    "    model_file=\"src/model.pkl\",\n",
    "    training_set=train_set,\n",
    "    framework=\"sklearn\",\n",
    "    label_column=\"label\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(import-enablemm-deploy)=\n",
    "### Import, enable monitoring, and deploy the serving function\n",
    "\n",
    "Use the [v2_model_server](https://www.mlrun.org/hub/functions/master/v2-model-server/) serving function \n",
    "from the MLRun hub. This function tests one or more classifier models against a held-out dataset.\n",
    "\n",
    "Add the model to the serving function's routing spec [`add_model()`](https://docs.mlrun.org/en/stable/api/mlrun.runtimes/mlrun.runtimes.html#mlrun.runtimes.ServingRuntime.add_model), \n",
    "enable monitoring on the serving function [`set_tracking()`](https://docs.mlrun.org/en/stable/api/mlrun.runtimes/mlrun.runtimes.html#mlrun.runtimes.ServingRuntime.set_tracking),\n",
    "and then deploy the function [`deploy_function()`](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.deploy_function)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-09-21 09:36:50,729 [info] Starting remote function deploy\n",
      "2025-09-21 09:36:51  (info) Deploying function\n",
      "2025-09-21 09:36:51  (info) Building\n",
      "2025-09-21 09:36:51  (info) Staging files and preparing base images\n",
      "2025-09-21 09:36:51  (warn) Using user provided base image, runtime interpreter version is provided by the base image\n",
      "2025-09-21 09:36:51  (info) Building processor image\n",
      "2025-09-21 09:38:36  (info) Build complete\n",
      "2025-09-21 09:38:44  (info) Function deploy complete\n",
      "> 2025-09-21 09:38:51,786 [info] Model endpoint creation task completed with state succeeded\n",
      "> 2025-09-21 09:38:51,787 [info] Successfully deployed function: {\"external_invocation_urls\":[\"tutorial-admin-monitoring-serving.default-tenant.app.vmdev91.lab.iguazeng.com/\"],\"internal_invocation_urls\":[\"nuclio-tutorial-admin-monitoring-serving.default-tenant.svc.cluster.local:8080\"]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://tutorial-admin-monitoring-serving.default-tenant.app.vmdev91.lab.iguazeng.com/', 'name': 'tutorial-admin-monitoring-serving'})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the serving function\n",
    "serving_fn = mlrun.import_function(\n",
    "    \"hub://v2_model_server\", project=project.name, new_name=\"monitoring-serving\"\n",
    ")\n",
    "\n",
    "serving_fn.add_model(model_name, model_path=model_artifact.uri)\n",
    "\n",
    "# enable monitoring on this serving function\n",
    "serving_fn.set_tracking()\n",
    "\n",
    "serving_fn.spec.build.requirements = [\"scikit-learn~=1.5.2\"]\n",
    "\n",
    "# Deploy the serving function\n",
    "project.deploy_function(serving_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Invoke the model\n",
    "\n",
    "Invoke the model function with [`invoke()`](https://docs.mlrun.org/en/stable/api/mlrun.runtimes/mlrun.runtimes.html#mlrun.runtimes.RemoteRuntime.invoke)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from time import sleep\n",
    "from random import choice\n",
    "\n",
    "iris_data = pd.read_csv(\n",
    "    \"https://s3.us-east-1.wasabisys.com/iguazio/data/iris/iris_to_predict.csv\"\n",
    ")\n",
    "iris_data = iris_data.to_numpy().tolist()\n",
    "\n",
    "model_name = \"RandomForestClassifier\"\n",
    "serving_1 = project.get_function(\"monitoring-serving\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your serving model function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'f3d62d27-b5f3-47ef-8ecd-03d618d1573c',\n",
       " 'model_name': 'RandomForestClassifier',\n",
       " 'outputs': [1],\n",
       " 'timestamp': '2025-09-21 09:38:53.028889+00:00',\n",
       " 'model_endpoint_uid': 'ddb24846b6f24bbba6410e98f3bc6ced'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_point = choice(iris_data)\n",
    "serving_1.invoke(f\"v2/models/{model_name}/infer\", json.dumps({\"inputs\": [data_point]}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After invoking the model, you can see the model endpoints and minimal meta data (for example, \n",
    "last prediction) in the **Models | Model Endpoints** page.\n",
    "\n",
    "<img src=\"../tutorials/_static/images/model_endpoint_1.png\" width=\"1300\" >\n",
    "\n",
    "You can also see the basic statistics in Grafana.\n",
    "\n",
    "(register-model-monitoring-app)=\n",
    "## Register and deploy the model monitoring app\n",
    "The next step is to deploy the model-monitoring app to generate the full meta data. \n",
    "Add the monitoring function to the project using [`set_model_monitoring_function()`](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.set_model_monitoring_function). \n",
    "Then, deploy the function using [`deploy_function()`](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.deploy_function).\n",
    "    \n",
    "This tutorial illustrates two monitoring apps:\n",
    "- The first application is a custom monitoring app (code provided).\n",
    "- The second application integrates with [Evidently](https://github.com/evidentlyai/evidently) and captures its results as MLRun artifacts.\n",
    "\n",
    "Learn how to write your own app in [Writing a model monitoring application](https://docs.mlrun.org/en/stable/model-monitoring/applications.html).\n",
    "\n",
    "After deploying the apps they show in the UI under **Real-time functions (Nuclio)**.\n",
    "\n",
    "### Deploying a monitoring app\n",
    "Set and deploy a monitoring application.\n",
    "\n",
    "If you're not using Iguazio's Jupyter, download {download}`demo_app.py <src/demo_app.py>`.\n",
    "\n",
    "```{admonition} Note\n",
    "You can also import model monitoring applications from the [MLRun hub](https://www.mlrun.org/hub/). Each application has complete usage instructions.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-09-21 09:38:53,076 [info] Starting remote function deploy\n",
      "2025-09-21 09:38:53  (info) Deploying function\n",
      "2025-09-21 09:38:53  (info) Building\n",
      "2025-09-21 09:38:53  (info) Staging files and preparing base images\n",
      "2025-09-21 09:38:53  (warn) Using user provided base image, runtime interpreter version is provided by the base image\n",
      "2025-09-21 09:38:53  (info) Building processor image\n",
      "2025-09-21 09:40:28  (info) Build complete\n",
      "2025-09-21 09:40:38  (info) Function deploy complete\n",
      "> 2025-09-21 09:40:44,164 [info] Model endpoint creation task completed with state succeeded\n",
      "> 2025-09-21 09:40:44,164 [info] Successfully deployed function: {\"external_invocation_urls\":[],\"internal_invocation_urls\":[\"nuclio-tutorial-admin-myapp.default-tenant.svc.cluster.local:8080\"]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://nuclio-tutorial-admin-myapp.default-tenant.svc.cluster.local:8080', 'name': 'tutorial-admin-myapp'})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_app = project.set_model_monitoring_function(\n",
    "    func=\"src/demo_app.py\",\n",
    "    application_class=\"DemoMonitoringApp\",\n",
    "    name=\"myApp\",\n",
    ")\n",
    "\n",
    "project.deploy_function(my_app)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To retrieve a list of the model monitoring functions, run [list_model_monitoring_functions](https://docs.mlrun.org/en/stable/api/mlrun.projects/index.html#mlrun.projects.MlrunProject.list_model_monitoring_functions).\n",
    "For example: `functions = project.list_model_monitoring_functions()`,"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploying an Evidently-based app\n",
    "You can use the MLRun built-in class, `EvidentlyModelMonitoringApplicationBase`, to integrate [Evidently](https://github.com/evidentlyai/evidently) as an MLRun function and create MLRun artifacts.  \n",
    "As a prerequsite, you need to have an Evidently workspace and project. If you don't have one, create it now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import uuid\n",
    "from src.evidently_app_setup import setup_evidently_project\n",
    "\n",
    "evidently_workspace_path = os.path.join(\n",
    "    mlrun.mlconf.artifact_path.replace(\"{{run.project}}\", project.name),\n",
    "    \"evidently_workspace\",\n",
    ")\n",
    "\n",
    "evidently_project_id = str(uuid.uuid4())\n",
    "setup_evidently_project(\n",
    "    evidently_project_id=evidently_project_id,\n",
    "    evidently_workspace_path=evidently_workspace_path,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now continue with registering the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2025-09-21 09:40:44,331 [info] Starting remote function deploy\n",
      "2025-09-21 09:40:44  (info) Deploying function\n",
      "2025-09-21 09:40:44  (info) Building\n",
      "2025-09-21 09:40:44  (info) Staging files and preparing base images\n",
      "2025-09-21 09:40:44  (warn) Using user provided base image, runtime interpreter version is provided by the base image\n",
      "2025-09-21 09:40:44  (info) Building processor image\n",
      "2025-09-21 09:43:00  (info) Build complete\n",
      "2025-09-21 09:43:20  (info) Function deploy complete\n",
      "> 2025-09-21 09:43:25,905 [info] Model endpoint creation task completed with state succeeded\n",
      "> 2025-09-21 09:43:25,906 [info] Successfully deployed function: {\"external_invocation_urls\":[],\"internal_invocation_urls\":[\"nuclio-tutorial-admin-myevidentlyapp.default-tenant.svc.cluster.local:8080\"]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://nuclio-tutorial-admin-myevidentlyapp.default-tenant.svc.cluster.local:8080', 'name': 'tutorial-admin-myevidentlyapp'})"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# register the second app named \"evidently_app\"\n",
    "\n",
    "my_evidently_app = project.set_model_monitoring_function(\n",
    "    func=\"src/evidently_app.py\",\n",
    "    image=\"mlrun/mlrun\",\n",
    "    requirements=[\n",
    "        f\"evidently=={SUPPORTED_EVIDENTLY_VERSION}\",\n",
    "    ],\n",
    "    name=\"MyEvidentlyApp\",\n",
    "    application_class=\"DemoEvidentlyMonitoringApp\",\n",
    "    evidently_workspace_path=evidently_workspace_path,\n",
    "    evidently_project_id=evidently_project_id,\n",
    ")\n",
    "\n",
    "project.deploy_function(my_evidently_app)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure alerts\n",
    "\n",
    "The monitoring infrastructure can trigger alerts when an event occurs. This alert occurs for the results on the model endpoints that are created by the evidently app."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the notification\n",
    "First define the notification. \n",
    "Define the notification list to send when the alert occurs. The example below creates a single Slack notification. Note that this is an illustrative example. In an actual implementation you need to create a slack webhook with a unique URL and pass that secret as a parameter.\n",
    "See full details in {ref}`notifications`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "notification = mlrun.model.Notification(\n",
    "    kind=\"slack\",\n",
    "    name=\"slack_notification\",\n",
    "    secret_params={\n",
    "        \"webhook\": \"https://hooks.slack.com/\",\n",
    "    },\n",
    ").to_dict()\n",
    "\n",
    "# Construct a list of notifications to be included in the alert config\n",
    "notifications = [\n",
    "    mlrun.common.schemas.alert.AlertNotification(notification=notification)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the alert object\n",
    "Create the alert configuration and store it in the project. The next time this event is detected, you will see an alert activation.\n",
    "See more details in {ref}`alerts`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alert_configs = project.create_model_monitoring_alert_configs(\n",
    "    # Name of the AlertConfig template\n",
    "    name=\"drift-alert-suspected\",\n",
    "    summary=\"user_template_summary_EventKind.DATA_DRIFT_SUSPECTED\",\n",
    "    # Retrieve metrics from these endpoints to configure the alert\n",
    "    endpoints=project.list_model_endpoints(),\n",
    "    # AlertTrigger event type\n",
    "    events=[mlrun.common.schemas.alert.EventKind.DATA_DRIFT_SUSPECTED],\n",
    "    notifications=notifications,\n",
    "    result_names=[],  # Can use wildcards\n",
    "    severity=mlrun.common.schemas.alert.AlertSeverity.LOW,\n",
    "    criteria=None,\n",
    "    reset_policy=mlrun.common.schemas.alert.ResetPolicy.MANUAL,\n",
    ")\n",
    "for alert_config in alert_configs:\n",
    "    project.store_alert_config(alert_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a unique ID for the event ID using the endpoint, application name and result name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the endpoint\n",
    "endpoint = (\n",
    "    mlrun.get_run_db()\n",
    "    .list_model_endpoints(\n",
    "        project=project.name,\n",
    "        function_name=\"monitoring-serving\",\n",
    "        model_name=\"RandomForestClassifier\",\n",
    "    )\n",
    "    .endpoints[0]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_name = \"MyEvidentlyApp\"\n",
    "result_name = \"data_drift_test\"\n",
    "\n",
    "# Construct the unique ID for the event ID\n",
    "result_endpoint = f\"{endpoint.metadata.uid}.{app_name}.result.{result_name}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate traffic against the serving function model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(1000):\n",
    "    data_point = choice(iris_data)\n",
    "    serving_1.invoke(\n",
    "        f\"v2/models/{model_name}/infer\", json.dumps({\"inputs\": [data_point]})\n",
    "    )\n",
    "    sleep(choice([0.01, 0.04]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View model monitoring artifacts and drift in the UI\n",
    "\n",
    "Now you can view the monitoring application results by clicking on the project Artifacts tab:\n",
    "\n",
    "<img src=\"../tutorials/_static/images/mm-myapp.png\" width=\"1000\" >\n",
    "\n",
    "**And if you've used Evidently:**\n",
    "\n",
    "<img src=\"./_static/images/mm-logger-dashb-evidently.png\" >\n",
    "<img src=\"./_static/images/mm-evidently-report.png\" >\n",
    "<img src=\"./_static/images/mm-evidently-suite.png\" >\n",
    "\n",
    "For more information on the UI, see [Model monitoring using the platform UI](https://docs.mlrun.org/en/stable/model-monitoring/monitoring-models.html).\n",
    "\n",
    "## View model monitoring artifacts and drift in Grafana\n",
    " \n",
    "Monitoring details:\n",
    "\n",
    "![grafana_dashboard_2](../tutorials/_static/images/grafana_dashboard_2.png)\n",
    "\n",
    "And drift and operational metrics over time:\n",
    "\n",
    "![grafana_dashboard_3](../tutorials/_static/images/grafana_dashboard_3.png)\n",
    "\n",
    "All of the Grafana dashboards are described in [View model monitoring results in Grafana](https://docs.mlrun.org/en/stable/model-monitoring/monitoring-models-grafana.html).\n",
    "\n",
    "## Batch infer model-monitoring\n",
    "\n",
    "You can use the batch function stored in the [MLRun hub](https://www.mlrun.org/hub/functions/master/batch_inference_v2/)\n",
    "to evaluate data against your logged model **without disturbing the model**, for example a one-time evaluation of new data.  \n",
    "\n",
    "See more in [Batch_inference](https://docs.mlrun.org/en/stable/deployment/batch_inference.html) and [Batch inference and drift detection tutorial](https://docs.mlrun.org/en/stable/tutorials/06-batch-infer.html).\n",
    "\n",
    "## Done!\n",
    "\n",
    "Congratulations! You've completed Part 5 of the MLRun getting-started tutorial.\n",
    "Proceed to [**Part 6: Batch inference and drift detection**](06-batch-infer.ipynb) to learn how to perform batch inference \n",
    "using a logged model and a prediction dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlrun-base",
   "language": "python",
   "name": "conda-env-mlrun-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.22"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
